Google Chrome Benchmarks - JavaScript engines compared (Chrome vs Firefox vs IE)

[Ray]

Google's new browser, Chrome was released to the public recently, and along with various usability and technical improvements, it aims to offer a more powerful JavaScript engine than existing browsers.

This is interesting to us here at Wrensoft because we have been struggling with the limitations of JS for some time. One of the options of Zoom is to offer a search engine function that will run off CD/DVDs and Intranets without the need of web servers or additional client-side software (such as ActiveX or Google Gears). This meant storing the entirety of the index data in memory, and defined in large ".js" files. The nature of a search engine is also computationally intensive, so improvements in the performance of the JS engine would be significantly noticable in our usage.

With this in mind, we benchmarked the 3 browsers in their performance to search through varying sizes of data sets. The browsers used are as follows (all 32-bit versions):

• Google Chrome Beta (0.2.149.27)
• Mozilla Firefox 3.0.1
• Microsoft Internet Explorer 7.0.6001

It should be worth noting that Chrome is still in Beta, while the other two browsers are RTM releases. However, it should also be noted that Google are notorious for leaving products in "beta" forever (think Gmail and Video), so it serves to mean very little.


The Tests
We ran a series of test searches on each data set. These test searches (A to F) were chosen based on their computational intensiveness and complexity of the search. They vary from single word searches to multi-word searches that include wildcard operators (like * and ?)


Benchmark Results

1. Searching a medium sized site or CD/DVD

• 5,000 pages and 2,714,034 total words (33,936 unique words)
• JS size: 31 megabytes

2. Searching a large site or CD/DVD

• 20,000 pages and 10,851,610 total words (36,558 unique words)
• JS size: 129 megabytes

3. Searching a larger site or CD/DVD

• 30,000 pages and 16,269,125 total words (38,121 unique words)
• JS size: 435 megabytes

Conclusion
These results show that in the context of a computationally and memory intensive JavaScript application (such as a search engine), Firefox 3 is currently the best peforming browser in both speed of execution and memory usage.

Google Chrome performs comparatively well with smaller data sets. But the difference (against Firefox) becomes much more significant when memory requirements are drastically increased.

Internet Explorer has the slowest and most limited JavaScript engine. It has much more stringent limitations in memory allocation and will refuse to load a script when these limits are hit. Previous benchmarks have shown that Firefox 2 used to have similar limits, but with Firefox 3, they seem to have extended these and are currently allowing for greater use of memory.


Notes
These benchmarks serve to illustrate the performance of these browsers in the use of one specific application. They are not necessarily indicative of their performances in all uses and different search engine implementations.

All tests values were obtained by performing the test 3 times and the average value was taken.

Memory usage values were obtained from Windows Task Manager and reflects the Working Set memory used by the process after performing the search. In the case of Chrome, the two processes are added together. All browsers are re-started before each test and no other web pages have been loaded before or during the instance of testing.

Zoom provides server-side solutions in PHP, ASP, and even C++ compiled binary CGI, for online searching. These options provide enterprise-level performance and can be used to search up to a million documents. However, these options require a web server and are not suitable for the practical demands of CD/DVD distribution. Our JavaScript solution caters for this usage specifically. More information on Zoom can be found on our website.


Test Hardware
All of the above tests were performed on the following machine:
Core 2 Duo E6750 @ 2.66GHz
5 GB RAM
Windows Vista 64-bit SP1

[Ray]

We received some questions in regards to how these benchmarks compare with others that have been posted on the web. Most notably in how Google Chrome is not performing as well as illustrated elsewhere.

What's interesting is that most of the benchmarks out there are measured with a static test suite. These only reflect the features and usages of JS that the test suite developers determine to be important. Our usage is a little different in that our application is a real-life example and also, that it is an application which is designed to run offline and thus use much more memory than most.

One of the most popular and commonly used test suite is SunSpider. It is worth noting that this is developed by the WebKit team (WebKit being the rendering engine in Safari and now, Google Chrome). It would seem likely that this test would place less emphasis on heavy memory usage and this is why the results are different. However, you can see in our results that Chrome was actually performing well with the small data set. It was only when memory requirements were increased heavily that it started to be outperformed by Firefox.

[tracymarx]

what about testing with the just released IE8 Beta2 ?

[Ray]

We'll do some testing with the IE8 beta in the near future if people are interested.